Summary

3-seed mean val_bpb: 1.0807 (std 0.0005) | ~15.8 MB | 8xH100 SXM | ~185s TTT eval

Merged SOTA (PR openai#1019, 3-seed mean): 1.88218 nats. This run: 1.82463 nats. Delta: -0.058 nats. Clears the 0.005-nat threshold. Track A (fixed predictor) — zero eval-time adaptation.

Results (3-seed)

Changes from Merged SOTA (PR openai#1019)

1. Discriminative TTT — per-block adaptive LR (Novel)

Pre-quant AdamW TTT with per-block learning rate scaling: early blocks get 0.3x base LR (preserve learned features), later blocks get 1.0x (full adaptation). Linear interpolation across 11 blocks. Combined with freeze=0 (all blocks trainable) and 10 epochs. Inspired by ULMFiT (Howard & Ruder 2018).

Nearest PR: openai#1306 (flat LR, freeze=2, 6 epochs). Different: graduated per-block LR replaces binary freeze, all blocks adapt at calibrated rates. Delta: -0.010 BPP vs flat-LR TTT.

2. Coprime-stride multi-shard data loader

Weighted random shard sampling with coprime stride. Delta: -0.003 BPP.

3. Config (QK_GAIN=5.0, WARMDOWN=4000, GPTQ damp=0.005)

Delta: ~-0.003 BPP combined.

Compliance (Track A — Fixed Predictor)

• No SLOT — no eval-time delta optimization
• No TTT during eval — all TTT before quantization, within training budget
• No n-gram cache — no eval-time statistics
• No eval-time adaptation of any kind — model frozen after training+TTT+GPTQ
• Standard autoregressive sliding-window eval (stride=64)

Reproduction

pip install flash_attn_3 --no-deps --find-links https://windreamer.github.io/flash-attention3-wheels/cu128_torch291/
torchrun --standalone --nproc_per_node=8 train_gpt.py

Credits

Base: PR openai#1019 (@abaybektursun). Pre-quant TTT: PR openai#1006. Coprime loader: PR openai#1184 (@icryo). Discriminative fine-tuning: ULMFiT (Howard & Ruder 2018). Freeze=0: @MatoTeziTanka (Issue openai#140).